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Skills! The reason for identifying the project as a "feasibility" effort 
stems from the paj^ticulac problems related to tKe "hational" scope of the 
project and. to fts focus .on "speaking and listening" skills. There are 
particular problems related to large-scale testing efforts which are dif- 
ferent from smaller scale testing and there are specific issues related to 
the assessment of communication competencies which are different from art ^ 
assessment of other educational areas. This paper specifically addresses 
issues related {o a national asse^ment of speaking-and listening skills. 
These<;^issues 'are relevant, at a more general level, to measurement of communi- 
cation competencies in-^ variety of scittings, e.g., in classrooms, in^social 
situat ionTT in employment settings.'^ « 

The purpose of the NAEP/SCATeasibility Project was to deve»lop stra- 
tegies for assessing communication competencies of elementary and secondary 
school students. It was important that the assessment instruments be de- 
signed for possible use in the- National Assessment o^ Educational Progress 
'a nationaf survey which measures actiievement of omportant^educational ob- 
jectives. ^Therefore, instrument development had to take i-nto account the 
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specific characteristics or the National Assessment as well as general con- 
cerns of educational testing. /'Jhqse requirements translated into three 
specific issues related to a large-scale assessment'of speaking and listen- 
ing skills, Thp fi/st problem. area deals with the identification of the, 
-"domain of coirinLpffcation competencies which would be assessed. The second 
involves the selection'of measVrement- strategics whfch would be utilized, 
The third covers sources of test bias, speci fi call ji. those related to cul- 
tural and situational differences. 

Domai n Speci f i cat i on 

The first task of the NAEP/SCA project was,io specify the domain of 

communication competencies which yvould be addressed by the assess^^nt. 

The requirements of a national survey necessitated the inclusion of a broad 

* 

range of speaicing and listening skills rather than those of a specific gom- 
muni cati on curritlJl'umV" Howwer, unl i ke^sorhe other educati onaT areas , the 7 
domain of communication competencies is not well charted* One of the -rea- 
sons for this situation is that speaking and listening skills are not 
covered /in school programs in a consistent way. In some ca's^s communica- 
tion .competencies are taught in speech courses, which are a part of the ^ 
standard or the elective curricuTum. In many schools aspects of speaking 
anjl listening are integrated into the required language, arts program. 
There are some instances ^here* there is no di^rect instruction provided in 
communication competencies. However, ft would be difficult to find schools 
where these skills are not addressed somewhere in instructional activities, 
for example in grgup discussion, ora#> reports , and formal -and informal , 
listening situations. 



. . . - \ t • 

.' • '1 1 

•Another reason for the lack of specification of tihe domain of communf- 
cation competencies is its orientation on skills rather^thanS knowledge/ 
^ The domain does not include many facts about speaking and lis^tening which * 
one might ejcpect students to know* Most 6f the.Vt)n^ain fofuses on skills. ^ 
Hoj/ever, it is difficult to ideifBify the skills \^hich ar0 widely accepted 
a|; central! Some communicati^ educators emphasize rhetorical skills re- 
lated to fonnal speaking situations. Others stress a rangfe of inforaal ' 
communication abilities whict] are important to school, socfal and employ- 
inent situations. The lack of consistent communication curricula anji the 
emphasis on a' va^riety of skills rather than specific f^cts made spe'cifying 
the domain of copiuncatvori competencies a difficult 'task. 

* i . 

Irrespective of the firoblems cited .abave, there are soine precedents 
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in the speech communicatj on discipline for specifying t hp' domain of com- 
murfication competencies. Jke other educational areas,* communication edu- ' 
cation was influenced by the movement toward- be*havi oral objectives- and 
accountability. .A concern with behavioral 'objectives <first>^surfaced in * 
the field of speech coirmuni cation in the4^te sixties -'(Baker ,-''•1967) and 
was promoted heavilyn'ri the early -seventies, -particularly by Kibler ' \ 
(Kibler, Barker and Miles, 1970; Kibler, Barker and CegaU', 1970a; ! 

Kibler, Barker and Cegala,^197Qb; Cegala, Kibler, Barker and MiTes, 1972)'i . 

. ^ ^ , • ' ; ' i ^ 1 ' 

One of "the resujts of this movement was the development of comprehensive , 

descriptjions of high school speech programs which included lists of be- 

tiavioral; objectives (Cortright, Niles afid Welrich, 1968; Buys, Carlson, 

Compton ^nd Frank, 1968; QlarJs: and Nelson, 1969). - ' ^ 

The; promotion of behavioral objectives in comnuni^iation education sQdn 



expanded to general support for the accountability movementV Many speech! 
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educators called for eva/uating the quality of s/eech programs in terms of 

/ ■ / / • ■/ . • 

studjint outcomes CYou/g, 1971; Findley. 1 912 ^Ratl i f f e . 1973). The thrust 

.for/ behavi oral/ o.bje/utives and' accountabilitiy^was countered by some speech • 

' copuni cation sch/lars who argued that t)ie conplexity and the.'human ele- 

' / ■ / ' ' / ■ / • ■ 

rti^nts of the coimuni cation process were difficult to describe in specific/ 

behavioral, te^n^s. CTucker, 1973; Aronoff, 1974; Streff, 1976; Xivi kly, 1976) 

,'Nevertheles/, the* re^Jlts of the m<^vement towarrfcl behavioral oBjective's and 

/ / • / ' • 

accountability provided a number'^of approaches for defining communi'cation 

abilitjr in terms of specific oDmpetenci^s , the latest example be/ng the 



.list of objectives developed by the Rational -Competencies Project from a 
questionnaire' study of speech communication educators and theorists .(Cegal a 
and Bassett, 1976). ' , 



/ 
/ 



Although the NAEP/SCA project considered a number of existinig and 
novel systems for specifying ^speaki ng and 'listening skills, it 'settled on 
the ca't-egories of functional communication competencies proposed by the . 
National Competencies Project (Allen and Brown, 1976) ao the basis for the 
domain description. These broad categories included fnforming, -controlling 
(persuading), sharing feelings, imagining and ritualizing/ Although, not 
entirely mutually^ exclusive,, the categories represented, a comprehensive 
.system for describing communication competencies which are important in a 
wide variety of formal and informal settings. 

The domain description was further defined by dividing each communi- 

cation function into speaking and listening perspectives*. Although^ the 

» primary emphasis 'of the description was on verbal communication colppeteri- 

* ' ' ■ i ' 

cies, the frarfiework als'O provided for nonverbal abilities as well. 



' . The third component of the domain description identified communication 

contexts. * These were categorized by formal and informal situations and by* 

dyadic and group situations/. The rules for appropriate communiication are 

fairly stable in formal comnunication situations. ^However, they* vary 

greatly irt informal situations/ depending upon the norms of the cultural 

group. Similarly, the rules for appropriate Communication are different 

depending upon the number of persons involved in the situation. The cqm- 

ponent ofi context was added to accoimodate these cultural and situational 

aspects of communication. v . .* 

. • 

In addition to the functional categories » an area/was specif i^od wh'ich 

« 

encompassed communication attitudes. This'^cpmponeat of the domain'iden- 

tlTted the predispositions toward conwunication iiehavibr which underlie 

the functions, perspectives and contexts. - , _ . ' 

/"The resulting Assessment Matrix (shown in Figure l)'^ncluded three 

dimensions: function (infoming, controlling, sharing feelings^ ritual iz- 

"^ing and imagining), perspective (speaking/expressing, listen] ng/recogniz- 
* ♦ 

ing) and context {formal dyad,*formal |roupi^ informal dyad, informal group) 
Attitudes were included as a separate component. \ *» 

Bas^d on th^ Assessment Matrix, the project further defined communi- 
cation competencies in tenns of specific objectives. At this po-int it was 
necessary to make some compromises. ^ Instead *of attempting. to chart the 
entire domain of speaking and listening skills,, the. project attempted to 
list some priority objectives which would be considered representative but 
not exhaustive of the^entire area. One additional compromise was made. • 
The project decided to exclude from the objectives and item development 
the area of imagining. This decisionrwas based upon two reasons. First, 



the imagining area,, which focused on the use of communication for creative 
purposes, considered to be substantively different from the*other cate- 
-gor1es-,-wh.tQh focusad^oji tlie -use of cormiuni cation for a number of f une- 
tional purposes. It was' felt that the imagining category wou-lfl requi^re 
unique measurement strategies and would*. detract from the majority of the 
development effort. Secondly, the creative use of 'communication was 
covered to some extent in existing National Assessment efforts. • For ex- 
ample, /aspects 6f language appreciation and oral reading abilities are 
assessed in the literature and reading suryeys. The List of Assessment* 
Objectives is provided io Figure 2. r^Also, due to tlie limited level of 
effort of the NAEP/SCA project, it was decided to focus on development of 
assessment strategies for 17--y6ar-olds. It was felt that an assessment of 
17-year-olds (one of four target groups of the NAEP) best repres.ented the 
culmination of ^the elementary and secondary educaftion process. 

Measurement Strategies 
A second problem for the NAEP/SCA project was the selection of nTea- 
surement strategies. The major issue telated'to this task revolved around 
the feasibility of direct versus Indirect measurement. Direct measurement 
of speaking typically requires collecting .samples ot speech performance 
'and scoring the samples according, to some sn>i5t of pre-^determined crjtei^iat 
Direct jneasurement of listening typically involves providing various spoken 
stimuli and asking multiple-choice questiojis whicji assess recall ^ inter- 
pretatj^on and evaluation of the material. Indirect ijieasurement of speak- 
ing and 'listening typically consists of multiple-choice questions which 
assess knowledge of appropriate connmuni cation behaviors in particular 



list of Assessment Object! \{es 



In^onninq/Speakii 




1. 

2: 
3. 



le to choose' an appropriate topic for 
an Infomative talk. 



Be able to introduce a topic In an informa- 
tive ta^^i. 

Bff able to» conclude effectively an infonna- 
tive talH. A . . - 

• 4, Be able to organize ap informative talk,--- 

5.* Be able to elaborate a point 

'6. Be able to ask question 

7. Be able t^ive JirQCtions. ^ 

8. fee a^l^to ^describe objects, people, places 
and' activities. ' 

-Cohtro 1 1 i nq/Sp5ak i nq . , 

1. Be able to use^ersuasi ve arguments. / 

. 2. Be able to support a position witlwfJpro- 
priate arguments. ^ • 

. 3. Be able to support arguments with appro- 
priate evidence. 

4* B'e able to analyze iin, audience. 

5. Be able to di^ti.r1guish facts from* opinions. 

6. 8e able to identify 'ogical 'iargu:n6nt>^ from 
"* illogical 'arguments. * . 

' Sharing Feelings /Expressing 

1. Be able' to recognize i f the situatioo- is 
appropriate for sharing feelings. ^ 

-2. ffe able to assert feelings in'^difficult 
communication situations. 

Ritualizing 

I. Be able to lead a group discussion. 
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Be able to identify ritual communications 

from other types of communicationT^ 
t 

3. Be able- to perfom typical communication 
rituals. * 

Communicating Attitudes 

!• Be willing to cormiunicate with others. 
Have a positive concept of self as a 



2. 

3. 
4. 



communicator. 

Be ]{/iTling to share feelings. 

Be comfortable communicating in difficult-^ " 
communication situations" (in front of* large 
-audiences, in groups, with authority figures 
or strangers). 



Informing/Li stenifiq * . ^ 

9. Be able to listen -actively'to and analyze an 
Informative speech. ; • 

to,. Be able to listen actively to an everyday 
Informative message. ^ 



Controlling/Listening ' ^ 

^ 7. Be able to listen actively to and analyze a 
^ persuasive speech. 

8* Be able to listen actively to an everyday per- 
^* ^ suasive mersage. 



/■ ■ 

Sharing Feel i ngs/Recogni zi ng 

3. Be able to recognize nonverbal expression^ of 

f typical feelings; r ' ,\ ✓ - 

4. Be able to recognize incotighrtnce between ver- 
bal and nonverbal expressions of typical feeli 



ngs< 



Be*comfortable when involved in persuasive 
coflimini cation. 

Be comfortable when it^vtfTyjd In cof^-nunicatioh 
rituals. , ^ 

Be tolerant of different dialects and ways of 
communicating. 



.8^ 
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situations. Listening is usually not measured using, \ndifect strategies. 

The >precedents for assessing functional conmunication have been re- 
ported by Larson, Backlund, Redmond and Bwbour (in prfess)* They renewed 
ninety instruments'-which represented both direct and indirect measures of* 
'productive and receptive abilities for all age groups. The tests were 
div;ided into seven^categories: 1) Developmental: Language and Comijiunica- 
tion Sj^.lls* 2) Communication Competence and Appropriateness, 3) Receiving: 
Listening^ 4) Receiving: MoTiverbal Sensitivity/Empathic Skijls, 5),Appre- 
hens'i on/Anxiety, 6) Interaction Descriptions, 7) Correlates: Disclosure/ " 
.Accessibility, Style and Preference, Attitudinal Correlate. This collec- 
tion focused on broad measures of the dimensions of communication competence 
It excluded mor^'e specific tests of communication skills, such as public 
speaking. ^ 

m ' , * * 

^Tests of specific speajcing ^ncl listening skills have been reported in ' 
»comnuni cation textbooks" and curricula (e.g., Allen, Anderson and Hough, 
J968; Brooks, and Friedrich, 1973;- Jeffrey and Peterson, 1975; Heun and 
Heun, 1975). However, 'these- tests rarjelV .have established norms, reli-. 
ability or "validity. Thus, there were only a few precedents for measuring 
specific functional communicatin^n competencies^ to guide the decisions of. . 
direct versus 'Indirett measurement strategies. 



The scope Qf the National Assessment of Educational Progress, which 
survey\ groups of^2,500 of students from all p^irts of the country, favored 
Indirect measurement strategies. It is much easier to assess large numbers 
of 'Students using multiple-choice type questions' than using^pey'formance 
tests; However,* the nature. e^4:he domain, which focuses more on skills " 
than specific facts, argued for the necessity of direct measurement » 

. . ..9-- 



strategies. In order to measure skills, it Is'more appropriate to give 
students actual tasks to perform rather than to give them multiple-choice 
questions. Without the guidance^ of prior instrument 'development, it 
seemed necessary that the NAEP/SCA project explore, at least on a feasi- 
bility ba^sis, the full range of 'possible measiirement strategies. There- 
fore, the project initiated development of indirect nTeasure's of speaking 
skills, direct measures of listening skills, and laid plans for direct 
measures of .speah'ng sk^ills. The following summarizes the problems faced 
by each of these development efforts and the results of initial project 
activities. * " 

. * - ' ■ ' 
^Indirect Measures of Speaking Skills 

There are two major problems for the development of indirect measures 
of speakiog ability. The first 'is the creation. of realistic items which 
were relevant to students from;all over the 'country. The second is the 
identification of cj early correct and incorrect answers. It was essen-\ 
tial that indirect measures of speaking skills be appropriate for all the 
students in the national sample and that they clearly represent competent 
and incompetent communication behaviors. 

fhe first issue related to. indirect measured of speaking skills is the 
generality of the items. This is a problem for any large-scale assessment 
instrument. It is necessary that students are tested on the intended know-- 
ledge and skills and not on' their familiarity with the testing situation. 
For example, jtems which lest speaking skills in the contesrt of subways, 
shopping centers or harvesters are not universally familiar. Furthermore, 
che dojnajn of coiniiuni cation competencies introduces .the confounding element 

-10- 
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/■ . ■ 

Of cultural and socio-economic dj^^rsity* Appropriate communication varies 
from group to group/ for exahlple, ways of sharing feelings differ depend- 
ing upon the customs of a j^rticular group. The problems of geography, 
.culture and socio-economic diversity made the development of indirect mea- 
sures of speaking. skil Is problematic* . . 

The setdn^ issue in developing indirect measures of speaking skills* 
revolves around response alternatives* It is difficult to write Inulti pie- 
choice questions with clearly correct and incorrect answers. Again, part 
of this problem stems frum the cultural and social natu»^e of commun^' cati on 
skills. Communication behavioV which is appropriate for one group is not 
necessarily appVopriate*for another group. This problem is accentuatecl " 
when communication moves froiii formal to informal situations and from, task 
related to -personal purposes. It is^possible to develop questions with^. 
cleirly correctand incorrect answers about formal communication situations 
such as a' job interview and for specific communication purposes such as . 
giving directions. However, i'f' become much more^difficul t'to develop 
items about informal communication situations such as conversations w4th 
peers and. for personal communication purposes such as rituals anu .xpress- 

4 

ing feel i'ngs . • "^-^ 

The NAEP/SCA project developed a variety of strategies for dealing 
with the issues of indir^ect measurement of speaking skills. Figures 3, 
4 and 5 provide examples of some of the different form'ats used to assess 
various functio{\^ of communications within various contexts. Figure 3 is 
an* example of a controlling (or persuading) task in two different typ^s 
of informal situations. Figure 4 is an example of two informing tasks* in 
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* ' • Figure 3 

i '** Individual 'Items " 

A man wants, to persu^de.,his colleagues to reconsider theiV support of 

^* ( ■ • 

Allqn Barber. Which of *^the following statements is the mosi logical 



argument he could use? 



"So-All en' Barber is .for freedom of' expression. I suppose he. 
^--^ thinks Ahat there's nothing v/rong i^i calling someone ginything 
^^you want to-, or advocating the overthrow of the government oY: 

yelling I'Firel * in a crowded building." » 

1 . ' ' , . ^ • 

. . "So Allen Barber" is /or freedom of expression. What does he' ' v 
^et^megn by that? Thertfare times- and places^where freedom of i 

expression may: be inappropriate. 'I thin|c we ought to" know 

spe.c,if1cal ly what he. thinks before we decide to support him."^- 

O I- don't know. '■ . * ' ■ ^ 



An advertising agency is trying to come up'with a way tcv sell more 
beer. Which one of the following slogans would be the most logical' 
argument they could use? * « ^ 

^ Or"Swing, Amen* can style; Drink Blue Bonnet Beer." 

CD "For a full,^ rich ta^te,-^ Drink Blue Bonnet Beer.".- 
. <0 I don*-t know, y I 



- ■ " « * F1qur6 4 - 

♦ 

* Related Items 

!• You have' been selected to host 'the awards ceremony for the Glendafe 
JOnlor Service Club.^^ Which would be the best way to ^tntroduce^the 
winner of the Outstanding Volunteer award? 

^^..^^^ "We 'al 1 know Nancy. She's been a member of the club for 
*CI^tmree years np^v. She^s a great kid. Nancy, come up and get , 

- the Outstanding Volunteer award." 

"The Outstanding VoJlunteer award is given each year to the ^ 
club member who demonstrates exceptional service and enthu- 
<3>siasm In club activities, This year's award goos to Nancy 

- Joyce fo;r her excellent job organizing this year's newspaper 
Kdrlve. Congratulations -Naticy.JU 

"This evening we wish to recognize Miss Nancy Joyce for her 
■noteworthy service in the Glendale Junior Service Club. She 
(3>waiS responsible fbr directing this year's successful news- 
fpaper drive. For this we present her with the Outstanding, 
Volunteer award. " . . ' 

0>I don't'lcncw. ' 

Z. At the end of the awards ceremony, which of the following statements 
would Be the best. way for you to conclude the-presentation? 



— * 



^^^^^"This concludes the Ninth Annual Glendale Junior, Service 
C^Clufa Awards Ceremony. It has b^en a\re*at honor for tne to 
be a part of this .important occasion." • 

"That's it. We have had a super year. I hope the next 
^^wiil be just as ^great. " 

"This marks the end of another'year for the Glendale Junior. 
^--^Service Club. We canTook back at our accomplishments' with 
^^pride and look forward to another productive and furi-filled 

year.". 

C^I donft know. 



14 



ERIC 



/ • . ■ 

V ' Figifre 5 

Multi-Part Items " ' ' 

1. D%:i let his friend Jack'^use his 10-speed bicycle. While Jack was 
stopping by a store, it was stolen. Jack tells Don* about the' s.itua- 
tibn between classes and says he is really sorry /it happened. But 
Don is' still angry. Would each of the following approac^^es be a 
' gooclway for Don to handle the situation? 



A. 'Don decides never to speak to Jack 
again, and holds his feelings inside* 



Yes No I don't know. 

O O O ' 



B» Don arranges to meet Jack during 

lunch so that he can tell Jack how <3) <0 ' 0> 

he feels about the whole thing* 

C. Don hassles Jack du^^ing the next 

break and doesn't give him a chance . ^"^^ ^\ 

to respund. , . 

D. Don decided to call Jack that night 

to discuss the whole situation when " <3) 
^ they bott} have plenty of time^ta 
talk. ' • / 
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a formal situation. Figure 5 is an example of a sharing feelings task in 

an Informal situation. - ^, • . 

* 

The results of the initial project efforts indicated that it was par-. 
• » • * 

tialljr successful in dealing with the issues. of dev'eloping indirect mea- 

^ ^/ ■ 

sures of speaking skills. The field test data suggested that the project 

»• 

was more successful dealing with^the task oriented communication- functions 
of informing and controlling than with the personal funptions of ritualiz- 
ing and sharing feelings. Furthermore, Items dealing with formal conimuni- 
Sation contexts were more adequate^^an items dealin^with informal con- 
texts. The need for an assessment which was appropriated to a diverse 
student population led to the creation of some items wuich were mundane- or 
irrelevant to students. The need for clearly correct and incorrect answers 
led to the development of some items which were too simple or 'obvious. 
Ei^re 5 provides an example of this type of problon. • 

Direct Measures of Listening Skills ^ ^ 

The measurement of listening skills appears on the surface to be less 
problematic than" the measuremertt of speaking sk i11s> There are "precedents 
for testing in this area. There are no problems in administering direct - 
measures of listening on a large scale. However; here too are several 
^•assessment issues. Probably the mosl^ cri'Jcal problem is to separate an 
assessment of listening ability from an assessment of -general verbal 
ability, since the research seems to indicate that these abilities are 
correlated (Barker, 1971). 

There are^ several standardized tests of listening skills (Brown and 
Carl sen, 1'J55; Educational Testing Services, 1957), ese tests typically 

. -15- -a 
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assess several .levels of listening skills, i»e», direct recall , 'interpreta- 
tion! and evaluation. In addition to these general approaches to assessing 
listening skills, speech communication educators and others have developed * 
tests of spec^^fic* analytical skills which are applicable in formal listen- 
ing situations^ These include such competencies as listening to a speech 
for organization, use of expository devices, use of evidence, fad- inference 
-distinctions and logical fallacies. 

The NAEP/SCA project cons4deriad {30th general and specific listening *' 
skills in its direct measures of .listening. It included listening compre- 
hension tasks arid speech analysis ta^ks. Furthermore, it sampled both * 
formal and infor/a> listening situations. For example, the stimuli in- 
eluded informative and persuasive speeches, a liewscast and a coimer^iial . 

The results of the initial development Activities ^surfaced an unex- 
pected probldm. in the^ direct assessment of listening skills. The listening 
items, more than any other group of items, elicited Significantly different 
responses from minority*and rronminority .students, ^umerous .explanations of 
this result were proposed: the level of vocabulary in .the -stimuli , the 
length of the stimuli ^, the rate and accent oi" the speakers on the stimulus' 
tapes, and distractions in the testing situat-^n. Probably tlie most plau- 
sible explanation is that the materials were^ied to general a&ility and 
the field testing did not control for these differences. Cilrrent'l'y, efforts 
are underway to reduce bias in these itans V sjippWfJTng vocabulary ('and 
thereby reducing some of the overlap between listening ability and general 

r 

verbal ability), shortening the stimuli and using interesting material., 

V 

It is essential that future field testing tn sdmeway: control for differences 
in general* verba-1 ability among the students. ' * . ' 



Direct Measures of Speaking Skills . 

The. most critical Issues for a large scale- assessment of speaking and 
listening skills revolve around the deve^lopment of measures, of speaking 
performance. A true measure of the domain of communication competencies, 
"Which focuses , primarily on skills rather than knowledge, must include^s ome 
direct assessment of speaking skills. And yet the national nature of the ' 
pilot project made this task particularly problematic. It is difficult 

o 

e to 'devise methods for a large-scale testing effort which will be feasible 
and alsOv objective. ' " , - ' 

- Due to the limited level of effort of the NAEP/SCA project, the issues 
-->^ related to the direct measures of speaking skills were only addressed. in a 
preliminary way. The problems were identified, preliminary strategies were 
proposed, but^ the f\tU scale development and field testing have not been 
conducted. These are the next critical steps for the Speaking and Listen- 
Ing , Assessment Project/,'The preliminary plans for direct assessment 
speaking skills are provided as .ah indication of future activity, ^ 
The assessment issues related to 'the direct assessment^ of speaking 
. performance are essentially the same as the issues related to the indirect 
assessment of speaking skills. * ^However, they require even more- attention. 
These problems include the development of communication tasks which are 
realistic -and relevant to students from all geographical, cultural and 
socio-economic groups and the establishment of objective crite>*ia for dif- 
ferentiating between competent and incompetent communication responses. 

I Two approaches ^were identified for dealing with the problem^ of mea- 
.suring speaking performance. -The first is based on assumptions about the 
' relationship between communication competence and; an individuals 

■ ,^ . , ^: ( 
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repertoire- of specific communication sMlls. In its most'natiiral form, 
' communication, competence can be defined as staying the right thing at the 
right time 'Cor sometimes keeping quiet). The judgment of appropriateness 
of the specific communication behavior is based on the particular social 
rules which. are operating in the situation. However, one can make the 
assumption that an individual is more likely to act competently more often. 
'If e or 5.he has a wi'il^epertoire of communication skillsWood, 1977a: 
'\1977b). ' 

Ttie aboye argument has led-the project to propose measuring communi- 

cation perfot^marrce in terms of the number and diyersity of speech stra- 

tegresAHse.drby a student in a specific situation. Students' would be asked 
J . . " - . :. ■ _ ■ 

to roTe play relevant 'communication tasks such as a job interview. The' 

speech samples would be analyzed in terms of the number and category of 

speech strategies utilized. The categories would generally follow' the 

functional category system devised by Wells (1973). This assessment 

approach w6lild provide a relat'ive measure of competence. ^Students who. used 

.m^e strategfes in more categories wouVd be" described as .more competent. 

A second way of measuring speaking performan<:e is based on the ability 

of the communicator to achieve his or her communication goals. For example 

if the purpose of a speaker is to give directions to the nearest drugstore, 

the measure of .success. would be the ability of the listener to filid the 

drugstore. 

Based on the above line of reasoning, tlje project^ proposes to set up 
specific coimunication tasks for students which can be judged in lerms of 
actual accomplishments. Using an established researcK technique ' 

(Brilhart, 1965) one student would be asked to describe a set of geometric 

if 
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figures to another student whb would be asked to draw the figurp. Success 
would be measured by the accuracy of the figures drawn by the second stu- 
•dent. '. • , " ■ 

Minority Test Bias 
The third assessment issue is so pervasive that it permeates the other 
two. This concern relates to minority test bias. It is essential that the 
assessment items measure actual communication competencies and not extran-^' 
* ecus factors, especially those which tap racial or ethrncji^^ 
is a particularly accute problem in an assessment" of^speaki ng and listening 
skills, because these competenG^-^f'^are^ tied to cultural and situa- 
tional , norms. In the. present project it was probably impos'SibVe to eliminate 
all .bias from an assessment. However, it was essential to recognize the 
bias and to cleanly indicate its'possible impacts. . , 

Several techniques »»ere used by-tfl^AEP/SCA project to deal with the 
issues of minority test bias. One major strategy was to establish a 
special Minority Review Panel. This group was asked to critique the assess- 
ment items in terms of their relevance to various groups of minority stu- 
dents and make suggestions for improvement. "The panel rewrote items to 
suggest contents or situations which would be relevant to minority students 
and also identified some items which could not be used aci:os,s cultural 
groups. For example, an outlining item was 'rewritten using the dining hours 
and types of food in Mexico as the' situation. Also, a, number of items des- 
cribing communication in dating situations were Identified as problematic 
because these behaviors., varied greatly across different cultural groups. 
It was felt that it was impossible to remove all bias within individual 
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items, that it was possible to reflect cultural dlveVs'ity clniong the ' 
items, thereby not providing one group with clear, advantages.* fhese assump- 
tions were'c^iecke^ out during the field testing ph^se of the project. 

The fiel^i testing^ of items was also directed toward detecting minority' 
test bias. The field. test sites n&re selected to include sizable popula- 
tions of Black and Hi spani c students. The analysis of the-*Vield test data" 
was designed to i\iclude correlations b^stween the resiJO^eV for each re- 
sponse choice orjfoil and ^he classi^fica't.ion of students as either minority 
or nonminority. Past experience in national assessments indicated that one 
might expect overall differences- in the ^performance of minont:? and aon- 
minority students in a wideTahge of content areas (Johnson, 1975). Thuj.-* 
one micjgt also ^expect the same in fuactional communitation comps^biges as 



we,n. However, it was important to eliminate factors which tended to'dis^ 
criminate between mfnority and n^nmino'Pity students but which were unre- 
lated to an assessment of functional communication competencies. The addi-" 

'■* 

tion of the correlational data provided a tool for identifying problem 
areas. , , ^ • 

0 

The results indicated that in, many cases the statistics were useful 
in^ identifying bias. Reviewers were able to examine the questions and find 
extraneous factors influencing the responses, such as speech styles, values, 
andvtackground experience. For exajiple,'one item-which measured fact- 
opinion distinct ions, -requested students to identify from a number of ^ 
s£atements the one t"hat was a fact. ' In field testing, minority students 
tended -to pick the response: "In order to get a decent job you have to 
finish school;"^ Although this statement was written as an' opinion, it may 
have tapped an important value of minority students. It appeared that the 
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value laden quality of the sta'lement may have confounded the assessment of 
the intendeV-CDmpetence. 

/. • • • X - • .. • 

In some# casei^he minority/nonminority distinctions were ^w^sle|ding. 
It appeared that^j^ racial/ethnic' factor was confounded by the level cf 
formal speech training.. One might expect that students* who had taken- speech 

^courses might perform better on some of the formal speech tasks, such as ^ 
P.r*epair1ng an outline fqr a speech. /"By chance the field test sites included 
one c^ll white school which had a required speech course at the tenth grade, 

. level and one all minority^ school whi^h offered no 'specific training in 
speech. This might .explain why the results of the outMhing'question dis- 
criminated between minority and nonmlnority students. However, this prob- 
lem may extend beyond the sample of- sites. Speech courses ate usually a 



part of ^he elective curriculum and are oriented toward college bound stu- 
Vdents. these factors suggest that more wli^'te students than minority stu- 
dents may be exposed to formal speech training. 

4 , 

The results of the initial project activities indicated that some of ' 

the Ttems selected still diffei^ntiated between minorities and .nonminori ties. 

' . • * * ' * • 

However, the review process eliminated a good deal 'of the test bias due to 
extraneous -factors. As indicated earlier, the problems with, the questions 
in the listening categories need investigating/ 

V 

Summary < it' 

The Feasibility Project for a National Assessment of Speaking and- 
Listening Skills, jointly initiated by the National Assessment of Educational 
Progress and the Speech Communication Association, represents the first" 
major effort in developing assessment instrument for functional comnuni-, 
cation competencies. The project had to address several difficult 
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methodological issues which are-^ peculiar to the large-scale scope of the 
assessment and it's focus on communication .competencies, including 1) spe- 
cifying the domain of communication competencies, '2) selecting measurement 

^strategie|, and 3) eliminating racial and ethnic bias. 

As might be expected in a feasibility effort, the NAEP/SCA project 
was partially successful in dealing with various methodological issues. 
The project developed adequate indirect measures of speaking skills and . 
direct measures of listening s kill's i and proposed two approaches for 

" direct Pleasures of speaking performance. The project was more successful 
in developing items for assessing functional con^^ categories of 

infoniiing and controlling (persuading) than, for the categories of ritualiz 
ing' and 'Sharing feelings. The project injplemented a number of/effective 
•techniques for identifying minority bias in items. Further work is neces- 

xsary-for the listening it4ms witi) respect to minority bias. Next steps in 
elude the refinement of current asse'ssment strategies and the full scale 
development and field testing of strategies for assessing speaking per- 
formance. 



1 . 
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